GH-50689: [C++][Parquet] Add IEEE-754 total order and nan count for floating types - #50807
GH-50689: [C++][Parquet] Add IEEE-754 total order and nan count for floating types#50807HuaHuaY wants to merge 2 commits into
Conversation
github-actions
Bot
added
awaiting review
|
There are many CI errors. I will fix them tomorrow. |
9b4091a to
5812e60
Compare
| ARROW_SUPPRESS_DEPRECATION_WARNING= \ | ||
| ARROW_UNSUPPRESS_DEPRECATION_WARNING= \ | ||
| GANDIVA_EXPORT= \ | ||
| PARQUET_DEPRECATED(x)= \ |
There was a problem hiding this comment.
Add this because I added a PARQUET_DEPRECATED at cpp/src/parquet/statistics.h and ci failed. https://github.com/apache/arrow/actions/runs/30975844655/job/92209544482
I believe this is a long-standing issue. If someone would like me to submit a separate PR to fix it, I can certainly do so.
github-actions
Bot
added
awaiting committer review
bd13067 to
f51ffc1
Compare
|
|
||
| int64_t null_count = 0; | ||
| int64_t distinct_count = 0; | ||
| int64_t nan_count = 0; |
There was a problem hiding this comment.
I would suggest directly using std::optional<int64_t> nan_count just like is_max_value_exact and is_min_value_exact.
There was a problem hiding this comment.
I choose to follow the style of null_count and distinct_count. I think there will be many diff changes to modifying them together. If we decide to do this, I recommend putting the changes in a separate PR. However, since it involves changes to a public API, we had better to put the two PRs in the same Arrow release.
2 participants
Rationale for this change
Implement IEEE 754 total order and NaN counts from apache/parquet-format#514.
What changes are included in this PR?
nan_countto statistics andnan_countsto PageIndex.Are these changes tested?
Yes.
Are there any user-facing changes?
cpp/src/parquet/types.h: AddsColumnOrder::IEEE_754_TOTAL_ORDER.cpp/src/parquet/properties.h: Adds the floating-point column-order writer property.cpp/src/parquet/schema.h: Allows column descriptors to use IEEE-ordered min/max statistics.cpp/src/parquet/page_index.h: Exposeshas_nan_counts()andnan_counts().cpp/src/parquet/statistics.h: Adds NaN fields and presence APIs toEncodedStatisticsandStatistics, and extends encoded-stateStatistics::Make/MakeStatisticsoverloads withnan_countandhas_nan_count.